1. Field of the Invention
The invention relates to a system and method for image and video reconstruction from multiple cameras. Specifically, the invention relates to generating user-selectable novel views on a viewing device that is remote from a server system.
2. Discussion of the Related Art
Multi-view reconstruction is the process by which a plurality of two-dimensional images of one or more volumetric objects is combined to create a single three-dimensional data representation of the objects. The representation can be rendered from any angle for a user.
Multi-view reconstruction is typically achieved using a point-cloud model derived from data that are extracted from the two-dimensional images and allocated to a three-dimensional virtual space. The computational challenge to build a point-cloud has been solved many times. However, known solutions have several drawbacks.
One significant drawback has been the inability to provide photo-realistic rendered output because known systems are unable to differentiate between objects that are being observed, e.g., foreground, and objects that are in the environment, e.g., background, to avoid the problem of occlusion.
A first type of occlusion is encountered when a virtual rendering camera is projected through an object that it should instead be projected on. The second type of occlusion occurs when a part of the object occludes itself, such as the hand of a player is in the foreground relative to virtual configuration and rendering camera and occludes a portion of the player's torso. When occlusion occurs and is not solved properly during rendering, it destroys a viewer's perception of the accuracy of the rendered output.
A further significant drawback is that colors are either entirely consistent, e.g., flat, or abruptly vary, e.g., jump, when the virtual rendering camera pans from one position to another. Such color issues are highly inconsistent with a viewer's normal perception. Photo-realistic results would radically extend the usability of the multi-view reconstruction to applications where currently humans are in proximity to the objects.
Thus, what is desired is multi-view reconstruction that provides a photo-realistic output that effectively solves occlusion and color problems.
What is also desired is generating user-selectable novel views on a viewing device that is remote from a server system.